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ABSTRACT 

This paper examines graphic computing environments, 
identifies potential problems in providing access to blind people, 
and describes programs and strategies being developed to provide this 
access* The paper begins with an explanation of how graphic user 
interfaces differ from character-based systems in their use of 
pixels, visual metaphors such as icons and windows, locational and 
contextual information, and mouse-controlled interaction and random 
access. The paper then analyzes how much of the benefits of the 
graphic user interface are scared by blind users. Three stages of 
access to the graphic user interface are described: (1) the 
customizing stage; (2) the ssingle-sensory mouseless strategy for 
providing access to standard text, icons and simple graphics, 
standard graphical structures, and navigation and control; (3) the 
multisensory approach with a resurrected mouse, designed to extend 
compatibility across applications and operating systems and extend 
access to complex graphics and other benefits. The paper concludes 
that, as problems are overcome, the resulting computer access systems 
will provide persons who are bl.^nd with new capabilities that were 
not possible with earlier character-based systems. (Includes 10 
references.) (JDJ) 
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THE GRAPHICAL USER INTERFACE CRISIS: 
DANGER AND OPPORTUNITY 

Abstract 

The graphical user interface (GUI) is both a powerful new interface for mainstream computer 
users and a source of serious concern for those who cannot see. Fortunately, graphic user interfaces 
can eventually be as made as accessible to blind people as any of their current character-based 
forerunners. In addition the interface systems developed to provide this access will also provide 
blind computer users with new capabilities that were not possible with the character-based computers 
and access systems. The effects of previous inaccessibility, the current accessibility limitations, and 
lingering doubts as to the solubility of some of the access problems has clouded the discussion and 
slowed efforts to capitalize on the advantages and opportunities that these new systems might bring. 
The purpose of this article is to examine graphic computing environments, to identify potential new 
problems posed by them, and to describe some programs and strategies that are being developed to 
provide access to these environments. 

ACCESS TO GRAPHIC USER INTERFACES BY PERSONS WHO ARE BLIND 

Our intuition tells us that the more an interface is optimized for a person who can cee, the less 
useful that computer will be to people who cannot see. Therefore, it is not surprising that many people 
reached the early conclusion that the new graphical computers (e.g. Apple Macintosh, Microsoft 
Windows, OS/2 Presentation Manager, UNIX X- Windows, etc) would eventually leave people who 
cannot see without adequate access to mainstream computer technology. In particular, it was widely 
held that blind people cannot make sense of what appears on a graphical computer screen, that they 
cannot interact with the computer using a mouse and pointing system, and that graphical computers are 
so different in basic design that even standard text cannot be understood. This in fact was the case for 
a jittery time when appropriate alternate interface technologies had not yet been developed. However, 
more recent work has shown that it is possible to design non-visual interfaces to windows, menubars, 
and pixel-based text (Boyd & Boyd, 1987; Thatcher 1990) . It has also been demonstrated that tactile 
displays, combined with speech output, can provide direct access to simple graphics, charts, and 
diagrams. (Vanderheiden, 1988, 1989; Vanderheiden & Kunz, 1990). Although these programs are 
yet in their infancy, they have demonstrated that people who are blind can use computers with 
graphical interfaces and can begin exploring new capabilities not available to them on strictly character- 
based machines, (e.g. drawing; reading flowcharts, schematic diagrams, maps, or floor plans; 
reviewing or creating musical scores; etc.) Thus, while the new graphical interfaces pose new 
problems for access by persons who are blind, they provide new opportunities as well. 
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This welcome paradox cannot be appreciated without more carefully examining the following 
questions: What makes the graphical user interface different from the old interface? What are the 
benefits of the graphic user interfaces that are problems to blind people? What is the current level of 
access to the graphic user interface, and what programs and suategies are underway to reach the ideal 
of full access to all of the new features and benefits of the graphic user interface? 

HOW IS THE GRAPHIC USER INTERFACE DIFFERENT FROM THE 
TRADITIONAL INTERFACE AND WHAT ARE ITS BENEFITS? 

The graphic user interface represents a fundamental change in the way computers display 
information and the way that humans interact with them. Figure 1 summarizes the major categories 
of differences from the old interface. The most technical and fundamental difference is screen 
rendering architecture. The use of pixels to put images on the screen leads to the access problem of 
deciphering information on the screen. The second major difference involves the way that people 
interact with the computer. This is divided into the way that the graphic user interface represents 
information on the computer display and the way that users manipulate and control ihe flow of 
information. This divides access problems into those that are primarily perceptual in nature and those 
that relate to program/computer control. 

(Figure 1 about here) 

Pixels as the Technical Foundation for the Graphic User Interface 

Fundamental to the graphic user interface is a basic change in the way that operating systems 
put information on the computer screen. Non-graphic computers (also called "character-based" 
computers) use a display text buffer to store information that is to be displayed on the computer 
screen. This buffer is a part of the computer's memory and has two numbers stored for each 
character position on the screen. One number represents the particular character for that position and a 
second number provides information regarding its color, boldness, underline etc. For example, 
uppercase "A", white on blue, is represented by the numbers (6531) in memory on an IBM 
computer. The first number (65) is the ASCII representation of "A", and the second number (31) 
encodes the color information. With this type of system one can cause a character to be displayed on 
the screen by simply putting the proper number into the proper place in the computers memory. 
Conversely, one can determine what character is displayed on the screen at any point by looking at 
the corresponding location in the computers memor> and reading the character (in A SCII) and its 
attribute information (color etc.) This type of system has the advantage of being very simple and 
straightforward. For access, it also has the advantage of providing a very easy way to access screen 
information. This display approach has the disadvantage that it cannot display proportional letters, 
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different sized fonts, or diagrams, charts or pictographic information. For people who are sighted 
this is a restriction that makes the computer display much less flexible than what is possible and 
widely used on paper. It also does not allow a person to see material on the screen in the same way it 
will look when it is printed out on paper. 

Pixel-based systems do not ha ve a text display buffer. Instead, the representation of the 
screen in memory is made up of pixels (dots). To cause a character to be displayed on the screen 
a program sets the proper dots in memory that are needed to form the desired letter shape. In 
effect, images are "painted" into the computers memory as clusters (shapes) of selectively 
darkened pixels. The computer then uses this dot information to create the image on the 
computer screen. Thus, we might have a cluster or arrangement of pixels that looks like a line, 
circle, or square; or we might have an arrangement that looks like an "A". The advantage of this 
approach is that it allows any size or shaped letter as well as any type of shape, line etc. As a 
result, graphic images, as well as printed information that will look just like the printed version, 
can be displayed on the screen. This approach provides for a graphic user interface which is 
easier for sighted persons to use, particularly if users are not completely familiar with a computer 
or with a program on the computer. The major disadvantage of the pixel approach to screen 
rendering is that it is more complex. More pertinently, since screen information is basically a 
pixel-based picture of the printed words etc., rather than a listing of the characters by position, 
this approach poses severe but surmountable problems for access by persons who cannot see. 

The Use of Visual Metaphors: Icons, Windows, Menus, Dialog Boxes, and Control 
Buttons 

The most radical of differences between the graphic user interface and the old interface is the 
representation or form in which information appears on the screen. Instead of using a highly 
specialized command language, the graphic user interface represents information as objects or visual 
images that are close to immediate tasks and everyday experiences. The primary device is the visual 
metaphor. Figure 2 shows a typical finder level screen for the Apple Macintosh based on an office 
metaphor containing a desktop (with scattered piles of lists, documents, and folders), two file 
cabinets (a hard disk and a floppy disk), and a trash container. The computer screen presented in 
Figure 3 typifies the use of visual metaphors in an application, in this case, a personal checking 
account program. This screen shows document metaphors of a bank draft, a transaction register, a 
directory of frequent payees, and a directory of expenditure categories. These document metaphors 
enhance communication by using objects that are familiar and directly pertinent to the tasks to be 
performed. 

(Figures 2 and 3 about here) 
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Locational and Contextual (Formatting) Information 

Often taken for granted by sighted people, objects on a computer screen have locational as well 
as intrinsic properties. That is, any object can be described in terms of its location on the screen 
relative tc the edges of the screen, relative to other objects, and relative to the pointer. For example, 
(as shown in Figure 2) the xash container is always located in the lower right hand corner of the 
screen unless you move it. The hard disk icon is immediately above the floppy disk icon. The Edit 
heading is located at the top of the screen, on the left, between the File and the View headings. This 
kind of information grcady facilitates finding something when you want it This is consistent with 
the way we tend to organize our daily work environments. 

The spacial arrangement of objects convey meaning in other ways too as in the formatting of 
documents, where paragraphs, columns, spacing, headings, indentations, footers, and other 
formatting devices convey meaning. Also, properties of degree are often communicated by an 
object's relative location on the screen and relational properties between objects such as relative 
importance, similarity, and belongingness are often communicated by the location of the objects 
relative to each other. 

Mouse Controlled Interaction, Screen Navigation, and Random Access 

The lower branch of Figure 1 identifies another feature of the typical graphic user interface that 
sharply distinguishes it from traditional interfaces. That feature is the way that it provides for user 
control and interaction between the computer and user. With early traditional interfaces, one had no 
choice but to learn the keyboard commands and procedures for each application used Commands 
and file names were typically typed again and again each time they were run or opened. Often, 
interaction with the machine required a tedious dialog of prompts and typed verbal commands. The 
graphic user interface takes an entirely different approach. A mouse is moved about a surface area 
such as a mousepad. Moving the mouse toward the right edge of the mousepad moves a pointer 
toward the right edge of the screen. Moving the mouse toward the bottom edge of the mouse pad 
moves the pointer toward the bottom edge of the screen. To select something on the screen, the 
pointer is moved over the object and the mouse button is clicked, rhis quick and direct access of 
programs, files, and commands is sometimes referred to as "random access." 

Commands and entries are r„iade die same way. For example, in the checking account 
example shown in Figure 3, a typical action would be to move the pointer over the 'Transactions" 
heading in the menu bar, pull that menu down (by holding down the mouse button), move the pointer 
over the desired transaction, and release the mous ; button. For check entries, you would move the 
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pointer over the appropriate blank space on the the check, click the mouse button, and enter the 
appropriate text, such as check number, payee, date, amount, and expenditure category. For 
recurrent check transactions, you can point to the payee's name on a list of frequent payees, click the 
mouse button, and have the vital parts of the check filled out automatically . Completed transactions 
can be reviewed in a check register. In like manner, checks can be cut, accounts reconciled, changes 
made, and transactions summarized on tables or graphs, all by pointing, clicking, and entering text 

The Standardized Interface 

The screen rendering capabilities of graphical computers facilitates the ideal of one interface for 
all applications. Increasingly, the tools for rendering graphical images on the screen and the tools for 
the mouse control of information flow are provided by the operating system Consequently, instead 
of designing and coding a new interface for every application, the application makers simply say, 
"Draw a menu bar here", "a window here", "a dialog box here", a "button hero", and so forth. The 
benefits of a consistent interface to developers are convenience and reduced development, 
documentation, and support costs. The great advantage of the standardized interface for users is that 
they do not have to start from scratch every time they try a new application. This same 
standardization of interface is possible with character-based screens, but because of the simplicity of 
screen access, most developers chose to go their own directions. In one sense the complexity of the 
graphic user interface has facilitated the use of a standard approach by developers. The standardized 
interface has critical implications for providing access to the graphic user interface by blind people. 
Specifically, knowing in advance how and where things arc done in the interf ace of most applications 
makes it possible and practical for third parry developers to tap into the flow of information in the 
graphic user interface and direct it to blind users through nonvisual channels. 

HOW GOOD IS CONTEMPORARY ACCESS TO GRAPHICAL COMPUTERS? 

The New and Only Meaningful Measure: How Much of the Benefits of the Graphic 
User Interface Are Shared by Blind Users? 

Only a short while ago there was no way for people without sight to access /japhical 
computers. Today there is. However, unlike traditional interfaces, it is not useful to think of access 
as an either/or proposition. What we really want to know is the degree to which specific access 
systems and new strategies provide blind people access to graphic user interface based computers. 
Specifically, we want to know which of the benefits of the graphic user interface identified above are 
shared by users who are blind and at wtuit levels. Also, as a longer range ideal, we would like to 
know how compatible an access system is across operating systems and how amenable it is to 
modification by third parties and individuals to respond to new situations and needs. 1* must also be 



JVIB Article 



5 8 



Boyd. Boyd, & Vanderheiden 



Access to the Graphic User Interface 



Thu, Sep 20, 1990 



kept in mind that these new access approaches need to provide access not just to textual information 
but to graphical (diagrams, charts, etc) information as well. This is information which was 
inaccessible with character-based access systems. Direct comparisons of access to graphic user 
interface computers with access to previous computers therefore can be somewhat misleading unless 
access to graphics on the text-based computers is taken into consideration. 

Having to meet both coverage and degree criteria, access systems theoretically could vary 
almost infinitely. However, for practical purposes, it is useful to think in terms of four general 
levels of access. This evaluation scheme is presented in Figure 4. 

(Figure 4 about heie) 
Stages of Access to the Graphic User Interface for the Blind 

In general, there have been three major periods or stages of development: a customizing stage, 
a single sensory stage, and a multi-sensory stage. This framework outlined in Figure 5 summarizes 
the progression of goals and strategies of the three stages. In brief, they evolve toward higher levels 
of access which, in general, tend to coincide with the levels of access described in Figure 4. Stage I 
and Stage II are based primarily on software only, on one adaptive aid (a screen reader or braille 
translator), and on one nonvisual channel (either speech output or braille). Stage HI is based on a 
multi-sensory approach with a combination of adaptive hardware and software. 

(Figure 5 about here) 

Stage I: The Customizing Stage 

Stage I access was not characterized by a particular specific system but rather a number of 
commercial applications having the same general strategy -- which was essentially a coping strategy. 
In these applications, only level 1 access is pursued (sec Figure 4) and most of the benefits of the 
graphic user interface were unrealized for blind people. In general, the sole purpose was to enable 
blind people in very specialized ways to read straight text produced on graphical computers through 
speech or braille. The problem which was confronted was the change in the screen-drawing 
architecture of graphical computers. Specifically, in the absence of a text buffer, conventional screen 
readers do not have readily available screen information in ASCII form to pass on to blind users via 
speech or braiLe. As one observer put it tersely only a short while ago: 

... the Macintosh uses a graphical interface that is essentially inaccessible to the blind: 
the Mac's screen memory contains only raw information about each pixel, and the 
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screen readers can't make sense of the bit-mapped display. The more graphical the 
interface, the less translatable it is into speech. A screen tull of icons, pictures, and 
overlapping windows becomes gibberish to a screen-reading program seeking clean 
ASCII code. To the extent that DOS and OS/2 emulate the Mac's graphical interface, 
the blind will soon be locked out of PCs as well" (Brody, July, 1989). 

Generally speaking, Stage I development consists of attempts to deal with the pixel problem 
by rewriting or modifying selected applications. The strategy was to selectively access information 
from within the program (where text information is known) rather than providing general access to 
the system or user interface. A good example of the thinking of this stage is a braille translator on the 
Apple Macintosh which consists of software that inputs standard text or ASCII files and translates 
them into braille. The obvious benefit of the translator approach is that it enables blind people to read 
documents produced by word processing applications. It can also be uswd with optical character 
recognition devices to translate paper documents into braille or speech. Therefore, it is valuable for 
providing blind persons access to documents produced on the Macintosh, and can be a valuable 
desktop publishing tool for braille libraries. Its most serious limitation is that it makes no attempt to 
interpret screen infoimation in real time or to actually use the Macintosh interface for control and 
navigation. As a result the user has access to portions of the output of the program but not to the 
system overall or to other programs. 

Stage II: The Single-Sensory Mouseless Strategy: 

Providing General Access to Standard Text 

The major objective of Stage II work was to provide level 2 access. Its central aim was to 
provide access to standard text across a wide number of applications. This aim became reality with 
the first general solution to the "pixel problem" which became known as the "interception strategy." 
Instead of trying to make sense of pixel-drawn information, this approach snares ASCII information 
before it gets to the screen. On pixel-based machines, readable ASCII information is provided by the 
application at the point where scrcsn writing services are called. This information is intercepted, 
modified, and stored in a special "off-screen buffer" before it is discarded by the screen writing 
services. This buffer is monitored and updated to provide specch-basea screen access. The first 
commercial application of the interception approach was outSPOKEN™ for the Apple Macintosh 
(Boyd & Boyd, 1987). 

Providing General Access to Icons and Simple Graphics 

In addition to access to straight text. Stage II efforts sought the ability to recognize icons and 
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simple graphics in standard applications. This required both technical and perceptual solutions in that 
the operating system must be able to identify the objects on the screen and communicate that 
information to people through nonvisual means. However, it is important to put this problem into 
perspective. 

One of the natural fears of people who are blind is that the trend in computers is toward total 
,, pictog^aphic ,, communication where visual metaphors completely replace words. In reality, the 
most predominant trend is toward pictures accompanied by words. This combination of pictorial and 
textual information provides more information and makes finding things easier, faster, and more 
reliable. For a tactile comparison, imagine all the groceries in a cupboard as having a braille label on 
them. Now imagine that all of the groceries (ketchup, beans, cereal, flour, etc) were all stored in 
cubes that were the same size and shape. As long as everything had a textual (braille) label, it could 
be eventually found and identified Storing different types of things in different places would also 
help. But finding things that differed only by their textual labels would be harder than if they also 
had differing shapes (ketchup in bottles, beans in round cans, cereal in different sized boxes, flour in 
a bag, etc.). Adding shape cues would provide searchers who were blind with additional cues which 
could make their searches faster, easier, and more accurate. The use of visual icons to accompjiny 
text labels can serve the same purpose for sighted users as tactile shape can for blind users. 

The use of icons is also not nearly as extensive as sometimes imagined. For example, all of 
the generic icons used at the finder level of the Macintosh can be counted on two hands (trash 
container icon, disk icon, document icon, program icon, window, close and resize box, and scroll 
bar). Almost all of these icons have word labels attached which can be read by interception-based 
access software. The rest can be easily recognized electronically and a name can be attached to them 

While a wider variety of icons is used to represent particular applications, it is not necessary 
for a blind person to perceive the shape of the icons to use them. All that the blind person needs to 
know is the fact that the pointer is over an object that represents an application whose name name is 
X. The names of the applications are readable with interception-based software. Simple graphs and 
charts are more problematic but they are just as problematic with character-based screen access 
systems where they are used. Fortunately, graphs and charts often complement rather than replace 
text Also, with spread sheets and data analysis applications, one often has the choice of having 
information summarized in a table containing text or in a bar graph or pie chart 

A frequent concern is how a speech-based screen reader can handle para-linguistic text such as 
fonts, italics, highlighting, underlining, and so forth. The interception-based software of Stage II 
recognizes and tracks this information and communicates it to the blind user by changing the pitch of 
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voice output when a change is encountered. If desired, the user can then get specific information of 
this kind by a designated keystroke. 

Providing General Access to Standard Graphical Structures 

An important event in Stage II development was the use the interception strategy of 
outSPOKEN to recognize and track information about windows, menus, dialog boxes, control 
buttons, and scroll bars. Information including type, status, size, location, and content of these 
standard graphical structures is stored in the off-screen buffer along with ASCII information about 
text, icons, and simple graphics. This information is conveyed to the blind user through speech 
when it is encountered by the screen review process of outSPOKEN or when solicited by the user. 
This capability is essential to the general access of graphic user interface computers across 
applications. While it is currently confined to the Apple Macintosh operating system, development 
on other operating systems is underway. 

Providing a Mouseless Navigation and Control System 

The approach to the so-called mouse problem taken in Stage H was to circumvent it That 
approach substituted the manual functions of the mouse with keystrokes on the number paa <n the 
keyboard. With the eighteen keys on the number pad, the blind user can perform most of the 
standand control functions of the mouse. This includes reviewing the desktop, reviewing the menu 
bar, selecting commands under menu columns, activating windows, moving the pointer 
incrementally across or down the screen, putting the pointer over desired objects, selecting them, 
dragging them, reshaping them, launching applications, reviewing, writing, and editing text 
Another keystroke pulls down a list of windows opened on the screen and another steps the pointer 
through the options. Others are used to activate windows, review the options in windows, select 
them, and launch them. This approach also provides a "find" keystroke that takes typed input and 
moves the pointer over the desired object To facilitate window management and between- window 
operations, it provides keystroke functions that resize and reshape windows to the user's 
specifications. 

Mimicking the Functionalities of Locational and Contextual Information 

The speech-based access software of outSPOKEN attempts to capture the functionalities of 
locational and contextual information in several ways. One is to verbally inform users when the 
pointer is moved to any of the screen's edges. Absolute screen location is provided by a function key 
that voices the screen coordinates of the pointer when it is pressed. In combination with a keystroke 
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that tells what is under the pointer, blind users can mentally pinpoint the location of objects for 
subsequent refereuce. Information about pointer and insertion bar locations within windows is also 
provided by keystrokes. This is particularly important when working with documents. outSPOKEN 
provides format information indirecdy through para-linguistic cues, such as the occurrence of bold- 
faced letters in headings, and through auditory cues such as beeps to signal the end of a line or 
cclumn of text More direct information includes the number of empty lines encountered between 
lines of text and the number of lines in a selected window. 

Locational information also comes with the standardized interface by learning the conventions 
of where objects usually appears. For example, the trash container cn the Macintosh desktop is 
always in the lower right corner (unless you move it). Another more indirect source of locational 
information comes with moving the pointer to the right or left and up or down and identifying what is 
under the pointer at various points. In this way, the blind user can reconstruct in his or her mind the 
screen locations of objects relative to each other. 

Major Shortfalls of Stage II Access 

Developments in Stage II fell short of the goal of providing blind people full access to 
graphical computers in six major ways: 

1) The interception strategy does not work with all applications. This is the result of 
incompatibilities between application software and recess software that occur when application 
developers write their own interfaces. The popular authoring application called HyperCard™ 
is a notable example of inaccessibility with systems based on the interception strategy. 

2) The speech-based strategy of outSPOKEN cannot interpret and commuRicate complex 
graphical information to blind people. 

3) In its present form, outSPOKEN cannot respond to new demands or be adapted by third 
parties or individuals to fit special needs and changing situations. 

4) The interception strategy is not easily applied to platforms other than the Macintosh. 

5) The single sensory strategy does not provide the full benefits of the scanning, browsing, 
memory jogging functions of the graphic user interface nor the full benefits of locational and 
formar* j information. 

6) The keyboard approach does not enable the blind person to use the mouse for navigation and 
control, for spacial location, and for recognizing complex graphics. 

Stage III: The Multi-sensory Approach with a Resurrected Mouse 

Extending Compatibility Across Applications and Operating Systems 

13 
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Two different strategies are being pursued in Stage III development to solve the 
compatibility problems of the first version of outSPOKEN. The most immediate and practical 
apptnach is to redesign the structure of outSPOKEN to make it modular, more generic in concept 
and structure, and more easily modified to meet new demands and special needs. New design 
criteria will more explicitly anticipate its role and integration in the multisensory- multimedia 
approach of Stage m. Among the specific goals is to make HyperCard and HyperText and other 
previously inaccessible applications accessible. A rt ited objective is to make it easier for third 
parties and end users to access the workings of access software to fit personal needs and 
unanticipated uses and situations. The strategy selected for accomplishing this is to enable 
HyperText scripting to be used to re-shape the access software. The long term objective of this 
effort is to lay the groundwork for compatibility across operating systems. 

The second major approach to solving the compatibility restrictions of earlier stages of 
development is to interpret information after it is rendered in pixel form. Theoretically this could 
be the ultimate solution to making access technology completely independent of platforms and 
operating systems. This potential is currendy being explored under the new name of VRT or 
"Video-image Recojniticn Technology" (Boyd & Boyd 1990). In brief, this technology seeks to 
adapt optical character recognition (OCR) memory scanning algorithms to interpret internally 
generated bit-mapped vi&o display information for standard text For more complex images it 
adapts and develops new AI irtiiicial intelligence) and pattern recognition techniques. This 
information is stored in off-screen text buffers analogous to the interception-based software. The 
advantage of this approach is that screen information can be interpreted regardless of what 
screen-rendering technology was used to put it on the screen. The downside is that practical 
applications are much further off than for the first approach. 

Extending Access to Complex graphics and Other Benefits of the Graphic User 
Interface 

Most of the shortcomings of Stage II development directly or indirectly involve the perceptual 
and navigation problems associated with the inadequacy of speech-based access alone. Stage m 
development seeks to capture the enhanced functionalities of the graphic user interface through a 
multi-sensory approach that integrates and expands technology developed for tactile-based access 
devices (Vanderheiden, 1988, 19b9; Vanderheiden & Kunz, 1990). In particular, it builds on the 
product called Optacon D™ by Telesensory Systems which directly presents a tactile image of text 
(and lines, graphics, etc) to a blind user through a set of vibrating pins. This approach avoids the 
problem of requiring a third party or the computer to interpret visual images because it leaves that task 
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to the user. This technology, fust developed with the optical scanning of documents in mind, was 
adapted to the transmission of pixel-based computer screens by the product called inTOUCH™. This 
software allows one to "feel" screen images on the tactile device as the mouse moves the pointer 
about the screen. 

The major problem with using Optacon II for computer access is that it ties up two hands - 
one on the mouse and one on the vibrating pin device. A second problem is that the mouse is a 
relative pointing device. That is, you use the mouse to tell a pointer on the screen to move up, down 
left or right from where it is but you get no tactile or kinesthetic feedback as to where you are 
pointing. To solve these problem, the Trace Center has developed a device that puts the vibrating 
pins of Optacon II on top of an absolute reference pointing device (similar in function to a graphics 
tablet puck). With this device, now called the "tactile puck/mouse," the blind person can feel under a 
fingertip the vibrating image of the pixel images that the pointer is passing over by the movement of 
the puck/mouse. The effect is a virtual full page tactile image of the screen. The resurrection of the 
puck/mouse to provide screen infonnation through a channel other than speech is a central ingredient 
in Stage HI efforts to increase the blind person's ability to interpret complex graphical images on the 
graphic user interface. It also restores more of the functionalities of scanning, browsing, and 
memory jogging, and thereby, more of the benefits of the graphic user interface. 

Another inadequacy of Stage II access that the tactile puck/mouse addresses is the loss of 
orientation, location, and context (formatting) information when a user is blind. Specifically, the 
problem with speech-based solutions only is that it is difficult to continuously keep track of where 
things are on the screen. The tactile puck/mouse (which also, on command, reads information aloud 
as it is touched on the screen) provides a quick way to explore the screen and provides the user with 
kinesthetic feedback as to location. With it the blind person can piece together mental images of 
where things are on the screen. This partially restored functionality is supported by another 
innovation that introduces the concept of haptic feedback (as a substitute for visual feedback). An 
application of this approach is the development of a new kind of mouse tablet that is made to 
correspond directly to the screen and a puck/mouse whose relative position on the tablet corresponds 
directly to the relative position of the pointer on the computer screen (Vanderheiden, 1988,1989). 
Thus, if the puck/mouse is at the upper left comer of ?he table, the pointer will be at the upper left 
comer of the screen. 

This haptic locational information can be further reenforced through the use of 3 dimensional 
sound reproduction (Boyd, 1989; Stevenson, 1989; Vanderheiden, 1989; Vanderheiden & Kunz, 
1990; Gehring & Morgan, 1990). With this approach, the words being spoken sound like they are 
coming from the location on the screen where they arc located. As you read from left to right, the 
voice seems to float from left to right If a word is heard coming from the right, you know you will 
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find it on the screen on the right edge. These audio location cues can either provide direct locational 
mrormation or reenforce haptic locational cues. 

To further accelerate input and control by users who are blind, the new interfaces being 
explored include tactile ribbings engineered into the surface of the virtual tactile tablet to provide 
points of reference between the edges and cardinal points. It is also outfitted with a row of "virtual 
buttons" then can be dynamically defined to provide fingertip availability of frequency used files and 
operations, macros etc. Multi-sensory systems are also under development that incorporate voice 
recognition for many of the repetitive user commands and operations (Vanderheiden, 1988,1989; 
Vanderheiden & Kunz, 1990). This frees up the user's hands and mind to concentrate on other 
things. But, the ideal is still to restore the functionalities of the mouse mat are not simply command 
functions. In particular, the ability to rapidly maneuver the pointer back and forth over and around 
objects on the screen with continuous (tactile, audio and haptic) feedback through the 
tactile/sonic/talking mouse/puck provides an important alternate nonvisual potential for interpreting 
pure and mixed graphics. It also provides the first opportunity for persons who are blind to explore 
drawing and other creative graphic activities. 

WHAT LIES AHEAD? 

It is about as certain as anything ever is in the computer business, that the graphic user 
interface is here to stay. It is also predictable that the graphic user interface will become even more 
dependent on vision as the industry moves toward multimedia live action and animation. An 
exa nple, is an interface glove that users will wear to move files, turn pages, punch buttons, and 
other such things. These will continue to pose new problems and require new strategies to overcome 
them. As the problems arise they v~U take the form of new barriers. As we overcome them 
however, the resulting computer access systems will provide persons who are blind with new 
capabilities that were not possible with earlier systems. For example, just as the systems discussed 
in this paper can provide a person who is blind with the ability to feel charts, floor plans, maps etc, 
newer systems based on 3 dimensional tactile and auditory displays may provide the ability to 
mentally construct the shape of a new car or other object "displayed" by a computer. When thinking 
about advances in computer architecture and accessibility by persons with disabilities it is useful to 
remember that the Chinese symbol for crisis is a combination of the symbol for danger combined 
with the symbol for opportunity. 
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Figure 1. Major Differences Between the GUI and Traditions: User Interfaces With 
General Categories of Features and Access Problems for Blind Users 
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Figure 2. Typical File Finder Screen of a Graphical User Interface 
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Figure 3. Typical Screen Within an Application Showing Windows, 
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General: Restricted to one operating system and one application. 

Difficult or impossible for third parties or individuals to adapt to new situations & needs. 
Specific: Can recognize and manipulate standard text in highly specialized and isolated situations. 



General: Restricted to one operating system. 

Applies to a wide range of applications. 
Specific: Can recognize and manipulate standard text. 

Can recognize icons and simple graphics. 

Can recognize pulldown menus, windows, popup dialog boxes, and buttons. 
Can move about the screen and manipulate screen objects (without the mouse). 
Limited ability to scan and browse screen information. 

Limited ability to perceive locations! and contextual (formatting) screen information. 



General: Easily adapted across operating systems. 

Applies to most applications and easily adaptable to special cases. 
Specific: Can interpret some standard complex graphics such as bar graphs and pie charts. 

Can use the mouse and pointer for orientation, navigation, and control 

Can scan and browse screen information. 

Can perceive locational and contextual (formatting) screen information. 



General: Applies across all operating systems. 
Applies across all applications 

Allows third party developers and individuals to modify to fit special needs and new situations 
Specific: Provides all of the functionalities of the GUI at the same levels as for sighted people. 



Figure 4. Levels of Access to the Graphic User Interface 
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Seeks Level 1 Access: Restricted to one operating system. 

Usually a customized approach for one application. 
STAGE I Limited to interpreting standard text 

Based primarily on software. 

Based primarily on one adaptive aid: Screen reader or braille translator. 
Based primarily on one alternate sensory channel: Speech output or braille. 



STAGE II 



Seeks Level 2 Access: Restricted to one operating system. 

Applies to a wide range of applications 
Interprets standard text, simple icons. 
Gin recognize and manipulate windows etc. 
Navigates by keystroke navigation system 
Limited browsing and format perception 

Based primarily on software. 

Based primarily on one adaptive aid: Screen reader or braille translator. 
Based primarily on one nonvisual sensory channel: Speech output or braille. 



STAGE III 



Seeks Level 3 Access : 
& 

Level 4 Access 



Maximum operating system independence. 
Compatibility with all applications. 
Can interpret graphs and charts. 
Can interpret complex drawings. 
Full browsing and locational capabilities 
Full mouse controL 
Supplementary voice input control. 
Easy modifiability by Third parties and individuals. 
Uses both software and hardware. 
Modifies and integrates adaptive aids. 
Integrates multiple nonvisual communication channels: 
Speech output. Voice recognition, Tactile & HapticsA Auditory cues 



Figure 5. Major Features of Access STAGES I, II, and III 
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